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FOREWORD 


Opinions,  interpretations,  conclusions  and  recommendations  are 
those  of  the  author  and  are  not  necessarily  endorsed  by  the  U.S. 
Anty . 


-  Where  copyrighted  material  is  quoted,  permission  has  been 

obtained  to  use  such  material. 

__ —  l^ere  material  from  documents  designated  for  limited 
distribution  is  quoted,  permission  has  been  obtained  to  use  the 
material . 


— _  Citations  of  commercial  organizations  and  trade  names  in 
this  report  do  not  constitute  an  official  Department  of  Army 
endorsement  or  approval  of  the  products  or  services  of  these 
organizations . 


JUL.  In  conducting  research  using  animals,  the  investigator (s) 
adhered  to  the  "Guide  for  the  Care  auid  Use  of  Laboratory 
Animals, ”  prepared  by  the  Committee  on  Care  and  use  of  Laboratory 
Animals  of  the  Institute  of  Laboratory  Resources,  national 
Research  Council  (NIH  Publication  No.  86-23,  Revised  1985). 


For  the  protection  of  human  subjects,  the  investigator (s) 


adhered  to  policies  of  applicable  Federal  Law  45  CFR  46 , 


.ILL—  In  conducting  research  utilizing  recombinant  DNA  technology, 
the  investigator ( s )  adhered  to  current  guidelines  promulgated  by 
the  National  Institutes  of  Health. 


JiJt—  In  the  conduct  of  research  utilizing  recombinant  DNA,  the 
investigator (s)  adhered  to  the  NIH  Guidelines  for  Research 
Involving  Recombinant  DNA  Molecules. 


aL 

the  investigator (s)  adhered  to  the  CDC-NIH  Guide  for  Biosafety  in 
Microbiological  and  Biomedical  Laboratories. 


In  the  conduct  of  research  involving  hazardous  organisms. 


3 


/ 


TABLE  OF  CONTENTS 

Page 

Cover  Page . 1 

SF  298  Report  Documentation  Page . 2 

Foreword . . . 3 

Table  of  Contents . 4 

Introduction . 5 

Body  of  Report . 8 

Conclusions . 21 

Schedule . 21 

Project  Staff. . 22 

Appendices 

Appendix  A:  Five  Modular  Response  Forms . 23 

Appendix  B:  Sample  Reports  of  Automated  Report  Writer . 38 

Appendix  C:  Automated  Reports  of  One  Case  with  Multiple 

Radiologists . 47 

Appendix  D:  Sample  Case  As  Listed  in  Database . 49 

Appendix  E:  Generation  Rules  for  Automated  Reports . 50 

Appendix  F:  Description  of  the  Grammar  Rules  for  Automated 

Reports . 56 

Appendix  G:  Instructions  for  Mammogram  Test  Readers . 60 


4 


/ 


INTRODUCTION 

The  aims  of  the  study  are  to  develop  a  computer-based  system  that  will  aid  die 
radiologist  in  interpreting  mammograms,  automatically  provide  a  standardized  report  of 
mammogram  findings  to  a  referring  physician,  and  construct  a  database  of  results  to  help 
assure  the  quality  of  the  interpretive  process. 

In  a  completed  system,  the  radiologist  will  assign  a  rating-scale  value  to  each  of  a 
set  of  perceptual  features  that  have  been  statistically  determined  to  be  diagnostically 
relevant  and  comprehensive.  Spoken  scale  values  will  be  recognized  by  the  system, 
merged  optimally  (in  terms  of  their  predictive  weights  and  intercorrelations)  to  yield  an 
estimate  of  the  probability  of  malignancy,  and  analyzed  interactively  to  generate 
automatically  a  prose  report  using  the  lexicon  of  the  American  College  of  Radiology.  A 
database  organized  about  the  perceptual  features  will  help  to  resolve  differences  in  dual 
readings,  to  construct  tutorial  materials  tailored  to  individual  radiologists,  and  to  adjust 
thresholds  for  recommendations  of  follow-on  imaging  or  therapy. 

Five  mammographers  at  Brigham  and  Women's  Hospital  (BWH)  have  assigned 
ratings  to  a  large  set  of  perceptual  features  for  200  proven  BWH  cases  both  to  determine  a 
necessary  and  sufficient  set  of  features  and  to  "train"  a  statistical  prediction  rule  to 
estimate  the  probability  of  malignancy  as  based  on  feature  ratings.  Five  radiologists  at 
Harvard  Pilgrim  Health  Care  (HPHC)  will  assign  ratings  to  features  in  a  set  of  150  proven 
HPHC  cases  to  "test"  the  statistical  prediction  rule.  The  mammograms  thus  interpreted 
at  HPHC  “  with  the  aid  both  of  the  list  of  perceptual  features  and  the  probability  estimate 
of  malignancy  --  will  be  compared  to  baseline  interpretations  of  the  same  cases  obtained 
there  earlier  from  the  same  radiologists,  in  order  to  determine  the  gain  in  accuracy 
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provided  by  the  computer-based  aid.  Reports  of  mammogram  findings  generated 
automatically  by  the  system  for  selected  cases  will  be  assessed  by  a  group  of  referring 
physicians  and  surgeons  relative  to  reports  dictated  for  those  cases  in  the  usual  way. 

This  project  builds  on  work  done  previously  in  the  BBN  laboratory  to  increase 
accuracy  and  extends  it  into  the  clinic  of  both  a  referral  center  (BWH)  and  screening  site 
(HPHC).  It  extends  previous  work  to  incorporate  the  reporting  process. 

The  statistical  prediction  rule  developed  earlier  led  to  significant  accuracy 
enhancements  and  has  now  been  refined  in  certain  ways,  principally  by  considering 
changes  in  the  perceptual  features  from  prior  to  current  mammograms.  The  linear- 
discriminant  analysis  used  earlier  as  the  technique  to  create  the  rule  was  replaced  by  the 
logistic-regression  technique. 

We  made  assessments  of  accuracy  by  ROC  analysis  (relative,  or  receiver, 
operating  characteristic)  to  obtain  an  index  of  accuracy  that  is  imaffected  by  an  observer's 
decision  threshold  and  by  the  relative  frequencies  (prior  probabilities)  of  malignant  and 
non-malignant  cases  in  the  test  set.  The  ROC  analysis  was  made  directly  from  the 
estimates  of  probability  of  malignancy  made  by  the  statistical  prediction  rule  and  also 
from  estimates  made  by  radiologist  observers  after  receiving  the  rule's  estimate  as  an 
advisory. 

Cases  were  obtained  retrospectively  at  the  two  clinical  sites  and  were  selected  to 
represent  malignancies,  benign  lesions,  and  "suspicious"  cases  that  were  determined 
subsequently  to  be  "normal."  Images  taken  at  two  different  times  were  included  —  the 
images  first  deemed  suspicious  and  the  images  of  the  last  preceding  examination. 
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The  radiologist  observers  are  representative  of  the  referral  and  community- 
hospital  settings,  respectively  —  the  former  being  more  highly  specialized  in 
mammography.  The  statistical  prediction  rule,  or  decision  aid,  is  thus  as  effective  as 
specialists  can  make  it,  but,  we  think,  still  suitable  for  the  different  case  mixes  of  various 
screening  settings. 

The  checklist  of  all  diagnostically  important  perceptual  features  is  an  aid  to  the 
radiologist  in  making  a  complete  assessment  of  image  information,  so  not  to  be  lulled  by 
a  premature  "satisfaction  of  search"  when  a  few  dominant  features  appear.  Following 
the  checklist,  however,  may  take  additional  time  and  the  radiologist  may  choose  to  use  it 
only  for  difficult  cases.  In  the  envisioned  practical  computer  system,  desired  cases  can  be 
selected  for  system  application  in  a  seamless  way,  interwoven  with  cases  not  selected; 
with  computer-based  speech  recognition,  the  microphone  usually  used  for  dictation 
controls  the  use  of  the  decision  aids.  Additional  motivation  for  using  the  decision 
system,  beyond  enhancements  of  accuracy,  include  the  automated  report  as  weU  as  a  rich 
database  of  diagnostic  findings  and  treatment  outcomes  on  all  cases  to  help  assure  quality 
in  several  ways.  Overall,  we  expect  the  cost-benefit  tradeoff  to  favor  system  use. 
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BODY  OF  REPORT 


The  Body  of  the  Report  that  follows  contains  a  section  on  each  of  the  main 
technical  tasks  accomplished  in  Year  3  of  the  project:  1.  Case  Selection;  2.  Checklist 
Development;  3.  Automated  Report  Writer;  4.  Data  Entry;  and  5.  Image-Reading 
Sessions. 

1.  Case  Selection 

As  outlined  in  the  original  workplan,  cases  have  been  qualified  and  entered  into 
the  study  from  two  sources  —  Brigham  and  Women's  Hospital  and  Harvard  Pilgram 
Health  Care.  The  Human  Research  Committees  of  both  institutions  have  approved  the 
study  protocol.  Cases  have  been  enrolled  in  3  categories  —  malignant,  benign,  and 
"suspicious  normal".  Tables  I  and  n  summarize  eligibility  criteria  and  the  enrollment 
statistics  of  the  final  case  sets. 


Table  1.  Eligibility  Criteria  and  Enrollment  Statistics 


Method 

of 

Final  Case  Sets  at 

Category 

Definition 

Proof 

BWH 

HCHP 

Malignant 

All  types  of  breast  cancer 
except  lobular  carcinoma-in-situ 

Pathology 

107 

50 

Benign 

Focal,  nonmalignant  processes 
(i.e.,  benign  tumors) 

Pathology 

53 

51 

Suspicious 

Patient  referred  for  additional 

Clinical/Imaging 

51 

49 

imaging  studies  or  accelerated 

(i.e.,  no  change  in 

follow-up  and  not  returned  to 

lesion  appearance 

routine  screening  pool 

monitoring) 

(Total  =  211)  (Total  =  150) 
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The  further  distribution  of  cases  over  types  of  mammographic  indication  are: 


Table  II.  Distribution  of  Cases  over  Types  of  Mammographic  Indication 


BWH  HPHC 


TYPE 

Suspicious 

Nonnal 

Benign 

Malignant 

Total 

Suspicious 

Normal 

Benign 

Malignant  Total 

Architectural 

distortion 

1 

1 

10 

12 

1 

0 

0  1 

Asymmetric 

density 

7 

4 

3 

14 

17 

1 

2  20 

Clustered 

calcifications 

5 

30 

35 

20 

10 

23 

17  50 

Regional 

calcifications 

3 

4 

6 

13 

0 

3 

0  3 

Mass 

35 

14 

53 

102 

21 

24 

31  76 

Patients  selected  fit  the  demographics  of  the  mammography  referral  (BWH)  and 
screening  (HPHC)  practices  at  our  two  sites.  This  fit  ensures  adequate  enrollment  of 
minority  groups. 

For  each  eligible  case,  all  available  original  mammographic  and  ultrasound 
images  at  the  time  of  the  "target"  examination  (i.e.,  when  the  suspicious  focus  was 
identified)  were  harvested  for  use.  In  addition,  in  order  to  support  development  and 
evaluation  of  "interval  change"  features,  mammographic  images  from  a  "comparison" 
examination  dating  approximately  12  months  (range  6  to  18  months)  before  the  target 
examination  were  also  pulled.  Patient-identifying  information  was  covered  by  removable 
tape  and  a  study  number  assigned  to  each  case  to  ensmre  patient  confidentiality. 

The  quality  of  images  in  each  case  enrolled  at  BWH  was  assessed  by  one  of  the 
BWH  investigators  (T.F.)  who  rated  overall  quality  on  an  ordinal  scale  (1  to  10). 
Likewise,  by  an  investigator  at  HPHC  (J.M.).  In  addition,  these  individuals  confirmed 
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that  all  needed  views  were  available  and  confirmed  selection  of  the  appropriate 
"comparison  study."  In  preparation  for  training  the  statistical  prediction  rule,  another  of 
the  BWH  investigators  (J.E.M.)  reviewed  all  the  selected  images  and  listed  the  coordinates 
of  the  most  suspicious  mammographic  abnormality.  This  step  ensured  that  the  expert 
readers  rendered  feature  ratings  on  the  same  lesion. 

All  available  clinical  mammographic  and  pathology  data  were  summarized  in  an 
electronic  relational  database  (File  Maker  Pro)  to  facilitate  data  extraction  for  tiie  study. 

2.  Checklist  Development 

It  remained  at  the  beginning  of  Year  3,  before  conducting  the  feature  assessment 
study  at  BWH,  to  convert  the  master  checklist  questionnaire  into  modular  form.  We 
created  a  separate  module  for  each  of  the  five  different  forms  of  radiographic 
presentation  of  the  lesions  that  occurred  in  both  the  BWH  and  HPHC  cases  samples:  1) 
Mass;  2)  Not  -Definitely-Benign  Calcifications;  3)  Asymmetric  Breast  Tissue;  4) 
Architectural  Distortion;  and  5)  Regional  Calcifications.  This  modular  arrangement 
enabled  the  reader  to  proceed  efficiently,  following  just  that  module  or  combination  of 
modules  that  applied  to  the  case  at  hand.  The  five  modules  are  attached  in  Appendix  A. 

3.  Automated  Report  Writer 

The  role  of  the  automated  report  writer  is  to  produce  a  written  English  version  of 
the  information  that  the  radiologist  has  entered  into  the  questioimaire.  The  reports 
should  sound  both  fluent  and  natural  and  they  should  stay  within  the  guidelines 
provided  by  tiie  American  College  of  Radiology. 

In  the  three  years  of  this  project,  we  have  successfully  built  a  protot5q5e  report 
generator  that  takes  as  input  the  data  from  the  questionnaire  and  produces  a 
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standardized  report.  In  order  to  target  the  work,  we  focused  on  one  t3^e  of  case,  those 
with  mass  findings.  However,  the  system  is  easily  extended  to  other  types  of  cases.  We 
produced  reports  for  over  five  himdred  cases  in  the  database. 

The  efforts  over  the  three  years  of  the  project  breaks  down  as  follows: 

•  Year  One: 

•  Analyze  the  complexity  of  the  problem  and  assess  the  different  approaches  to 
text  generation  and  how  they  apply  to  this  problem. 

•  Work  along  with  those  creating  the  questionnaire  to  ensiure  that  the  information 
necessary  for  the  report  will  be  elicited. 

•  Year  Two: 

•  Collect  actual  reports  and  compare  the  information  in  them  with  the  information 
in  the  questionnaire. 

•  Analyze  the  textual  variations  and  compare  them  with  specifications  in  the 
BiRads™  document  "Breast  Imaging  Reporting  and  Data  System." 

•  Design  an  architecture  for  the  report  writing  system. 

•  Year  Three: 

•  Implement  a  prototype  of  the  automated  report  writer  and  test  it  on  actual  data 
produced  by  the  system. 

•  Provide  reports  to  radiologists  for  feedback. 

The  work  of  the  first  two  years  of  the  project  is  described  in  the  previous  annual 
reports;  we  focus  here  on  summarizing  the  architecture  of  the  system  and  describing  the 
implementation  and  testing. 

We  first  siimmarize  the  final  system  and  the  results. 

Generated  Reports 

The  reports  are  generated  in  three  parts,  the  introduction,  findings  and 
conclusion.  The  introduction  includes  not  only  the  heading  with  the  case  number  and 
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data,  but  also  a  summary  of  the  history,  in  particular  whether  the  examination  is  with 
respect  to  a  previous  examination,  and  information  on  the  overall  condition  of  the  breast. 

The  second  paragraph  of  the  report  describes  the  findings.  Each  item  in  the 
questionnaire  is  incorporated  into  a  sentence.  In  some  cases,  multiple  items  are 
combined,  such  as  the  size  and  location  of  the  mass.  Other  items  are  rendered  as 
complete  sentences,  such  as  the  presence  of  microlobulation  or  other  conditions.  The 
conclusion  indicates  the  likelihood  that  the  findings  are  malignant. 

An  example  of  an  automatically  generated  report  is  shown  below.  The  database 
input  for  this  paragraph  is  included  in  Appendix  D.  We  also  include  a  set  of  30 
paragraphs  selected  randomly  from  the  case  set  in  Appendix  B,  and  a  set  of  six 
paragraphs  that  are  generated  from  different  readings  of  the  same  case  in  Appendix  C. 

Case  #118  2/11/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  5/31/1990.  The 
breast  is  almost  entirely  fat.  The  tissue  has  a  low  density. 

There  is  a  new  11  mm  irregular  mass  with  some  evidence  of  tissue  invasion  located 
at  the  approximate  3  o'clock  middle  position  of  the  right  breast.  A  small  portion  of 
the  margin  is  indistinct  due  to  tissue  invasion.  About  half  of  the  margin  is  obscured 
by  glandular  tissue.  About  half  of  the  margin  is  clearly  circumscribed. 
Microlobulation  is  present.  The  mass  may  be  an  intramamary  node. 

There  is  a  85%  likelihood  that  the  mass  is  malignant. 

As  we  describe  in  more  detail  in  the  next  section,  the  paragraph  is  generated 

from  a  set  of  rules  that  take  into  consideration  all  of  the  different  features  and  their  values 
in  the  database  that  report  on  masses.  The  rules  can  be  easily  changed  to  make  minor 
changes  in  the  text  of  the  report. 
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The  process  of  determining  the  actual  wording  is  done  in  three  stages,  the  first 
two  of  which  are  complete: 

1.  Analyze  the  wording  in  reports  written  by  radiologist,  which  was  provided  by 
Brigham  and  Women's  Hospital. 

2.  Work  iteratively  with  the  BBN  team,  including  our  domain  experts  to  ensure 
that  the  wording  accurately  expressed  the  information  in  the  questions  and  the 
choice  of  boimdary  points  is  correct,  for  example  if  the  value  on  a  1  to  10  scale 
is  4,  the  wording  might  be  "a  small  portion",  but  if  the  value  is  8,  the  wording 
might  be  "most  of  the...". 

3.  Get  feedback  from  radiologists  by  (1)  giving  radiologists  sample  reports  and 
questionnaires  and  asking  them  to  "grade"  them,  (2)  asking  radiologists  to 
write  a  report  after  filling  out  a  questionnaire  and  comparing  it  to  the  automatically 
generated  report. 

System  Architecture 

The  main  goal  of  the  system  design  was  to  use  the  simplest  possible  technology 
that  would  solve  the  problem  in  a  robust  and  portable  way.  Since  the  database  is 
completely  prespecified  (that  is  there  will  be  no  new  fields  or  values  while  the  system  is 
in  use)  and  the  goal  of  the  resulting  paragraphs  is  clarity  and  consistency,  rather  than 
creativity,  we  decided  to  use  a  simple  "direct  replacement"  grammar  approach.  There 
are  two  advantages  to  this  approach: 

•  All  of  the  information  specific  to  the  domain,  such  as  the  order  of  the  different 
parts  of  the  report  and  the  specific  words  to  be  used,  are  maintained  in  a 
declarative  set  of  rules  independent  of  the  code  that  runs  to  produce  the  report. 
This  make  it  both  easy  to  make  small  modification  to  the  report,  such  as 
changes  in  wording  or  additions  of  other  information  and  to  move  the  system 
to  whole  new  domains,  either  within  mammography  or  to  new  areas. 

•  The  system  runs  very  fast,  since  at  each  point  it  only  needs  to  select  among  a 
small  number  of  alternatives  and  the  computation  required  to  make  the 
selection  is  usually  just  a  lookup  in  the  database  and  comparison  of  number 
(e.g.,  is  the  value  less  than  four,  or  between  five  and  seven,  etc.). 

The  major  disadvantage  is  there  is  a  limit  in  the  sophistication  of  the  text  one  can 
generate.  For  example,  since  there  is  no  link  between  different  parts  of  the  report  or  even 
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between  different  sentences,  the  wording  is  often  repetitions.  For  example,  the  two 
sentences  in  ihe  above  report  "About  half  of  the  margin  is  obscured  by  glandular  tissue. 
About  half  of  the  margin  is  clearly  circiunscribed."  would  be  more  fluent  if  combined: 
"About  half  of  the  margin  is  obscured  by  glandular  tissue  and  the  other  half  is  clearly 
drciimscribed." 

We  decided  to  approach  the  architecture  by  using  the  simplest  possible 
mechanisms  for  the  first  prototype  to  imderstand  how  far  they  would  go  and  then 
incrementally  add  complexity  where  it  is  most  needed.  Too  often,  overly  complex 
software  is  used  to  solve  a  simple  problem,  which  adds  to  both  development  and 
maintenance  time.  We  would  like  to  complete  a  full  round  of  evaluations  with 
radiologists  before  moving  forward  on  increasing  the  system's  complexity. 

The  Generation  Grammar 

The  grammar  rules  contain  all  of  the  information  on  how  the  report  should  be 
organized  and  what  the  wording  should  be  in  the  individual  sentences.  A  rule  consists  of 
a  nonterminal  that  will  be  expanded  (called  the  "left  hand  side"  of  the  rule),  and  a  set  of 
one  or  more  expansions  for  that  nonterminal  (called  the  "right  hand  side").  For  example, 
the  first  rule  in  the  system,  shown  below,  has  the  nonterminal  $REPORT  as  its  left  hand 
side  (nonterminals  are  preceded  by  the  special  symbol  $)  and  then  one  right  hand  side 
option  for  expansion. 
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(de£rule($REPORT 

>  $INTRO  $FINDINGS  $CONCLUSION))' 

Each  nonterminal  on  the  right  hand  side  is  a  nonterminal  on  the  left  hand  side 
somewhere  in  the  grammar.  The  system  recursively  expands  each  of  those  nonterminals 
as  it  processes  the  paragraph,  so  for  example  $INTRO  is  expanded  into  $CASE  $DATE 
$HISTORY  SCOMPOSmON,  as  shown  in  the  rule  below,  which  again  has  only  one 
alternative. 

(defrule  ($INTRO 

>  $CASE  $DATE  $HISTORY  $COMPOSmON)) 

In  some  cases,  the  system  needs  to  insert  a  value  from  the  database.  So  in  the 
expansion  of  $HISTORY,  the  system  prints  the  words  "The  present  examination  is 
compared  to  a  prior  mammogram  of "  (words  are  in  quotes)  and  then  gets  the  actual  date 
from  the  database.  Note  that  this  rule  also  has  a  condition,  that  is,  it  only  prints  the 
sentence  if  there  is  in  fact  a  value  in  the  database  for  the  previous  date  (DATE_P). 
Otherwise  the  rule  is  not  expanded. 

(defrule  ($HISTORY 

>  "The  present  examination  is  compared  to  a  prior  mammogram  of " 

(: VALUE  DATE_P) ".  " 

:CONDmON  :EXISTS  DATE_P)) 

The  nonterminal  $COMPOSITION  expands  to  two  nonterminals, 
$GLANDULAR  and  SDENSITY.  Each  of  these  has  several  alternatives,  each  conditioned 
on  the  value  in  the  database. 

(defrule  (SCOMPOSITION 

>  $GLANDULAR  $DENSITY)) 


'  To  make  the  example  clearer,  we  have  left  out  text  formatting  information  from  the  rules  shown  here.  A  complete  list  of 
the  actual  rules  is  in  Appendix  E. 
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The  conditions  in  the  $GLANDULAR  rule  look  at  the  result  of  the  question 
"Percentage  of  tissue  tiiat  is  glandular,"  which  is  a  value  between  0  and  100.  The  rule  has 
four  alternatives  to  choose  from,  each  results  in  a  full  sentence  being  added  to  the  report. 
Note  that  if  one  wanted  to  change  either  the  range  for  each  choice  or  the  particular 
wording,  one  would  only  have  to  change  this  rule. 

(defrule  ($GLANDULAR 

>  "The  breast  is  almost  entirely  fat.  " 

:CONDmON  iBETWEEN  (:  VALUE  OV02_C)  0  20 

>  "The  breast  is  largely  fat.  " 

:CONDmON  iBETWEEN  (:  VALUE  OV02_C)  21 49 

>  "The  breast  is  largely  fibroglandular.  " 
iCONOmON  :BETWEEN  (:  VALUE  OV02_C)  50  79 

>  "The  breast  is  almost  entirely  fibroglandular.  " 

:CONDmON  :BETWEEN  (:VALUE  OV02_C)  80 100)) 

System  Implementation 

The  report  writing  prototype  is  implemented  as  a  stand-alone  system  that 
generates  reports  "off  line,"  that  is,  the  information  in  the  database  was  saved  to  an  ascii 
file  that  was  then  input  to  the  report  writer,  which  produced  all  of  the  reports  in  batch 
mode.  The  prototype  is  written  in  Lisp  and  nms  on  a  Sparc  Ultra,  but  it  could  be  easily 
ported  to  a  PC  using  a  platform-independent  language  such  as  Java. 

We  tested  the  system  by  running  it  on  531  cases,  which  the  system  processed  in 
imder  10  seconds  per  case.  Since  all  of  the  domain-specific  information  is  encoded  in  the 
grammar,  the  same  core  system  will  generate  reports  of  any  kind,  as  long  as  the  form  of 
the  input  is  similar.  This  means  that  the  system  can  easily  be  extended  to  other  finding 
types  and  other  types  of  radiology  reports  with  no  changes  to  the  computer  code. 

Future  Directions 

We  have  completed  a  prototype  system  capable  of  producing  reports 
automatically  from  the  database.  The  next  step  is  to  get  feedback  on  tiie  readability  of  tiie 
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reports  and  whether  the  wording  accurately  captures  the  intentions  of  the  radiologists 
when  they  put  a  particular  response  in  the  questionnaire.  When  they  say  that  the 
percentage  of  tissue  that  is  glandular  is  45%  would  they  also  write  that  "The  breast  is 
largely  fibroglandular?"  Also,  what  is  the  most  effective  way  of  showing  the  experts  the 
rules  so  that  they  can  easily  make  the  appropriate  changes  to  wording?  We  have 
experimented  with  producing  a  text  version  of  the  expansion  of  all  of  the  rules  (see 
Appendix  F).  Is  this  easier  to  xmderstand  than  the  raw  rule  format? 

Another  important  area  is  the  integration  of  the  prototype  with  the  full  system, 
which  would  allow  the  reports  to  be  produced  directly  after  die  questionnaire  is  filled 
out.  This  would  require  porting  to  the  PC  platform  and  integrating  more  tightly  with  the 
actual  format  of  die  database  being  used. 

Finally,  we  need  to  reassess  whether  this  most  simple  form  of  text  generation 
technology  is  actually  powerful  enough  for  the  application,  and  if  not,  we  need  to  determine 
which  kinds  of  extensions  will  provide  the  most  improvement. 

4.  Data  Entry 

In  Year  2  we  constructed  a  data-entry  program  to  permit  radiologist  readers  to 
enter  their  responses  to  the  checklist/ questionnaire  by  speaking  them.  In  Year  3,  we 
determined  that  even  Release  2  of  the  Phonetic  Engine  500  speech-recognition  system  was 
not  adequate  for  use  in  the  clinical  setting,  primarily  because  the  recognition  of  digits  — 
our  primary  data  —  was  not  sufficiently  reliable.  In  the  process,  the  program  was 
modified  to  accept  keyboard /mouse  entry  instead  of  voice.  It  was  further  modified  to 
permit  research  assistants  to  enter  orally  data  that  had  been  recorded  on  the 
questiormaires  in  written  form,  to  give  us  experience  with  the  speech-recognition  system 
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without  impacting  radiologists.  A  further  program  was  written  to  permit  research 
assistants  to  record  radiologists'  responses  on  a  lap-top  computer,  for  use  with  the 
enhanced  reading  at  HPHC.  The  program  calculated  the  statistical  estimate  prediction 
rule's  estimate  of  probability  of  malignancy  on  each  case  for  immediate  feedback  to  the 
radiologist. 

5.  Image-Reading  Sessions 

BMi-Readings 

Five  radiologists  at  BWH  read  200  cases,  assigning  a  value  to  each  of  66 
perceptual  features  for  each  case.  These  data  were  the  basis  for  two  statistical 
prediction  rules  developed  by  a  stepwise  logistic-regression  procedure  --  one  for 
masses  and  one  for  clustered  calcifications.  The  features  selected  for  each  prediction 
rule  are  shown  in  Table  m. 


Table  III.  Features  Selected  for  Prediction  Rules 


Rule 

Features 

For  Masses 

Shape  of  mass 

Percent  of  margin  that  is  clearly  circumscribed 

Size  of  mass:  computed  ratio  (maximum  size/minimum  size) 

Density  of  mass  relative  to  surrounding  glandular  tissue 

Patient  age 

Size  of  mass:  computed  change  (current  study  -  prior  study) 

Presence  of  related  architectural  distortion 

Presence  of  worrisome  calcifications 

For 

Calcifications 

Presence  of  related  architectural  distortion 

Patient  age 

Percent  of  tissue  that  is  glandular 

Change  in  size  of  focal  distribution  over  time: 

computed  ((current  study  -  prior  study )/years  between  studies) 

Degree  to  which  the  distribution  can  be  characterized  as  segmental 

Degree  to  which  the  distribution  can  be  characterized  as  linear 

Degree  to  which  elements  can  be  characterized  as  fine  linear 

Degree  to  which  elements  can  be  characterized  as  pleomorphic 

18 


The  performances  of  die  pooled  readers  and  the  two  statistical  prediction  rules 
are  given  in  Table  IV  in  terms  of  the  ROC  accuracy  index  (which  varies  from  0.5  to 
1.0). 


Table  IV.  Performances  (A,)  of  Readers  (Pooled)  and  Statistical  Prediction  Rules 


Malignant  vs. 

All 

Malignant  vs. 

Benign-Biopsy 

Cases 

Suspicious  Normal 

Readers 

.875 

.936 

.962 

MASSES 

(N~510) 

Statistical 

.881 

.943 

.966 

Prediction 

Rule 

Readers 

.725 

CLUSTERED 

CALCM 

CATIONS 

(N~350) 

Statistical 

.739 

Prediction  Rule 

As  an  ancillary  analysis,  we  measured  the  performances  on  the  cases  with  a  mass 
for  perceptual  features  taken  from  ultrasormd  imagery,  a  modality  usually  used  only  to 
determine  if  masses  are  cysts.  As  shown  in  Table  V  with  comparative  figvues,  the  novel 
result  is  that  a  statistical  prediction  rule  for  the  malignant/nonmalignant  distinction  that 
is  based  only  on  ultrasound  features  performs  very  well  as  a  diagnostic  tool  for  that 
distinction. 


Table  V.  Performances  of  Ultrasound  Features  A^ 


Readers 

.939 

Statistical  Prediction  Rule 
for  cases  with 
mammography  features 

.981 

SPRfor  cases  with 

ultrasound  features 

.925 

SPR  for  cases  with 

both  types  of  feature 

.983 
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HPHC  Baseline  Readings 


Baseline  (unaided)  readings  of  150  cases  were  made  by  five  radiologists  at 
HPHC,  in  anticipation  of  aided  readings  in  Year  4.  The  individual  performances  are 
given  in  Table  VI  in  terms  of  three  measures:  A^,  true-positive  proportion  at  false¬ 
positive  proportion  =  0.5,  and  positive  predictive  value. 


Table  VI.  Three  Measures  of  HPHC  Baseline  Reading  Performance 


Reader 

TPP@FPP  =  .50 

PPV 

1 

.85 

.90 

.47 

2 

.91 

.97 

.49 

3 

.84 

.89 

.47 

4 

.82 

.88 

.47 

5 

.84 

.93 

.48 

Mean 

.85 

.91 

.48 

With  based  on  pooled  (vs.  average  data),  the  Table  VII  gives  a  comparison  of 
HPHC  performance  on  (1)  all  cases,  (2)  masses  alone,  and  (3)  clustered  calcifications  only. 
The  same  comparison  is  given  for  BWH  performances. 

Table  VII.  Pooled  Values  of  A,  for  HPHC  and  BWH  Readers 


All  Cases 

Masses 

Calcifications 

HPHC 

.83 

.85 

.78 

BWH 

.86 

.94 

.73 

The  written  instructions  for  the  HPHC  baseline  readings  are  given  in  Appendix  G. 
Enhanced  Reading  Study  at  HPHC 

All  of  the  main  preparations  for  conducting  the  enhanced  reading  study  at  HPHC 
have  been  completed.  We  have  conducted  an  initial  briefing  session  for  the  readers  as  a 
group  to  go  over  the  general  procedure  for  the  reading  sessions.  This  group  session  also 
included  training  on  several  of  the  calcification  features  that  we  have  learned  in  prior 
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studies  need  some  close  explanation  and  illustration.  The  training  images  and  feature 
data  were  taken  from  cases  employed,  and  readings  generated,  in  the  BWH  study.  We 
are  presently  involved  in  physically  assembling  and  hanging  ihe  50-case  set  for  the  first 
of  three  enhanced  reading  sessions  and  are  scheduling  the  readers  for  sessions  beginning 
in  early  January.  We  anticipate  completing  data  collection  and  analyses  by  the  end  of 
March. 

CONCLUSIONS 

The  tasks  scheduled  for  Year  3  were  accomplished,  including  final  assembly  of 
the  BWH  and  HPHC  case  sets;  readings  by  BWH  readers  for  training  five  modular 
versions  of  the  statistical  prediction  rule;  construction  of  an  initial,  flexible,  extensible 
version  of  an  automated  report  writer;  computer  programming  for  non-voice  data  entry; 
baseline  readings  by  HPHC  readers;  and  initial  training  for  enhanced  readings  by  HPHC 
readers. 

The  project  is  proceeding  successfully.  A  proposal  to  the  U.S.  Army  Medical 
Research  and  Materiel  Command  —  to  install  the  mammography  decision-aiding  system 
on  the  World  Wide  Web  —  was  invited  by  the  Army  and  submitted  17  November  1997. 
SCHEDULE 

The  enhanced  readings  by  HPHC  radiologists  are  to  be  conducted  in  January  to 
March,  1998,  and  the  data  should  be  analyzed  within  the  next  month.  A  statistical 
prediction  rule  based  on  a  neural-network  approach  will  be  constructed  and  compared 
with  the  existing  logistic-regression  version.  Focus  groups  of  referring  physicians, 
oncologists,  and  surgeons  will  be  conducted  for  preliminary  evaluation  of  the  automated 
report  writer.  The  final  report  of  the  project  is  expected  to  be  prepared  on  schedule. 
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Appendix  A:  Five  Modular  Responses  Forms 


Reader  No. _ 

Case  No. _ 

Finding  No.. 


Response  Form— X-Ray  Mammography 


•  Identify  the  finding  FNDGl 

O  Mass 

O  Not-definitely-benign  calcifications 
O  Asymmetric  Breast  Tissue 

O  Architectural  Distortion 
O  Regional  Calcifications 
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Module  I 


Mass  (MM) 

Relationship  to  Prior  Study 

•  This  mass  finding  is: 

O  new 

O  not  significantly  changed 
O  significantly  changed 


Reader  No. _ 

Case  No. _ 

Finding  No.. 


MM20 


If  significantly  changed,  also  rate  the  prior  images  where  requested. 
Otherwise,  rate  only  the  current  images. 


•  Density  of  mass  relative  to  surrounding  glandular  tissue 


Current  o  i 

2 

3 

4  5  6 

7 

8 

9  10  MM04 

mass  density 
much  lower 

isodense 

mass  density 
much  higher 

•  Confidence  about  the  presence  of  fat  within  the  mass 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10  MM05 

definitely  NONE 
present 

definitely  some 
present 

Size  of  mass 

Current : 

Largest  diameter  (in  either  CC  or  oblique  view) 

Smallest  diameter  (in  either  CC  or  oblique  view) 

mm 

MM06 

Current : 

mm 

MM07 

Prior: 

Largest  diameter  (in  either  CC  or  oblique  view) 

Smallest  diameter  (in  either  CC  or  oblique  view) 

mm 

MM06 

Prior: 

mm 

MM07 
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Mass  (MM)  -  cont. 


•  Shape  of  mass 

Current  o  i 

2 

3 

4  5 

round/oval 

lobular 

Prior  0  i 

2 

3 

4  5 

•  Percentage  of  the  margin  that  is  clearly  circumscribed 


8  9  10  MMIO 

irregular 

8  9  10 


Current _ %  MM13A 

Prior  _ % 


•  Confidence  that  at  least  a  small  portion  of  the  margin  is  spiculated 


Current  0123456789  lo  MM12 


definitely  NOT 
spiculated 


definitely 

spiculated 


•  Confidence  that  the  mass  is  an  intramanunary  node 


Current  oi  23  45  67  89  lo  MM16 


definitely  NOT 
an  intramammary  node 


definitely 

an  intramammary  node 


•  Confidence  regarding  the  presence  of  related  architectural  distortion 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10  MM17 

definitely  NOT 

definitely 

present 

present 

•  Confidence  regarding  the  presence  of  worrisome  calcifications  within  the  mass 


Current  01  23456789  lo  MM18 


definitely  NOT 
present 


definitely 

present 
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Mass  (MM)  -  coni 


Inirial  Overall  Diagnostic  Tudgment 


•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chances  in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _  MMRAl 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 


Computed  Probability  of  Malignancy: 


Final  Overall  Diagnostic  Tudgment 


•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chzmces  in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _ ^ _  MMRA2 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 
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Module  II 


Reader  No. _ 

Case  No.. 


Finding  No.. 

Calcifications  (Not-Definitely-Benign)  (NC) 


Relationship  to  Prior  Study 

•  This  not-definitely-benign  calcifications  finding  is: 
O  new 

O  not  significantly  changed 
O  significantly  changed 


If  significantly  changed,  also  rate  prior  images  where  requested. 
Otherwise,  rate  only  the  current  images. 


NC21 


Element  Characteristics 


•  Size  of  largest  individual  element  (best  visual  estimate) 


Current 

o 

o 

o 

NCOS 

less  than  0.5  mm 

0.5  mm  to  1,0  mm 

more  than  1,0  mm 

•  Variability  of  size  of  elements 

Current  o  i  2 

3 

4 

5 

6 

7 

8 

9  10  NC06 

low  variability 
of  size 

high  variability 
of  size 
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Calcifications  (Not-Definitely-Benign)(NC>  -  cont 


•  Degree  to  which  the  elements  can  be  characterized  as  fine  linear 


Current  o  i  2 

3 

4  5 

6  7 

8  9  10  NC07 

definitely  NONE  of  the 
elements  are  fine  linear 

at  least  one  or  two 
elements  definitely  are,  or 
several  probably  are,  fiine  linear 

Prior  0  i  2 

3 

4  5 

6  7 

8  9  10 

Degree  to  which  the  elements  can  be  characterized  as  branching 

Current  o  i  2 

3 

4  5 

6  7 

8  9  10  NCOS 

definitely  NONE  of  the 
elements  are  branching 

at  least  one  or  two 
elements  definitely  are,  or 
several  probably  are,  branching 

Prior  0  1  2 

3 

4  5 

6  7 

8  9  10 

Degree  to  which  the  elements  can  be  characterized  as 

pleomorphic  (heterogeneous) 

Current  o  i  2 

3 

4  5 

6  7 

8  9  10  NC09 

definitely  NONE  of  the  at  least  one  or  two 

elements  are  pleomorphic  (heterogeneous)  elements  definitely  are,  or  several  probably  are, 

pleomorphic  (heterogeneous) 


Prior  0123456789  10  *  Degree 

the  elements  can  be  characterized  as  punctate 

Current  01  23  45  67  89  lO  NCll  defmitclv 

definitely  NONE  of  the  at  least  one  or  two 

elements  are  punctate  elements  definitely  are,  or  several  probably  are, 

punctate 


Distribution  Characteristics 


•  Number  of  elements 

Cturent  0 

0 

o 

NC13 

less  than  5 

5  to  10 

more  than  10 
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Calcifications  (Not"Definitely-Benign)(NC)  -  coni 


•  Size  of  the  focal  distribution 

Largest  dimension  in  CC  view 

Current  _ mm 

Prior  _ mm 


•  Degree  to  which  the  distribution  can  be  characterized  as  linear 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10 

definitely  NOT 

definitely 

linear 

linear 

Prior  0  i 

2 

3 

4 

5 

6 

7 

8 

9  10 

•  Degree  to  which  the  distribution  can  be  characterized  as  segmental 


Current  o  i 

2  3  4  5 

6  7  8  9  10 

definitely  NOT 

definitely 

segmental 

segmental 

Prior  0  1 

2  3  4  5 

6  7  8  9  10 

Relationship  to  Other  Aspects  of  This  Studv 

•  Confidence  regarding  presence  of  related  architectural  distortion 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10 

definitely  NOT 

definitely 

present 

present 

•  Confidence  regarding  presence  of  related  mass  or  asymmetric  breast  tissue 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10 

definitely  NOT 

definitely 

present 

present 

NC14 

NC14 


NC17 


NCI  8 


NC19 


NC20 
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Calcifications  (Not-Definitely-Benign)(NC)  -  cont. 


Initial  Overall  Diagnostic  Judgment 
•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chances  in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _  NCRAl 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 


Computed  Probability  of  Malignancy: 


Final  Overall  Diagnostic  Tudgment 
•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chances  in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _  NCRA2 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 
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Module  III 


Reader  No. _ 

Case  No. _ 

Finding  No.. 


Asymmetric  Breast  Tissue  (AT) 


Relationship  to  Prior  Study 

•  This  asymmetric  tissue  finding  is:  ATIO 

O  new 

O  not  significantly  changed 
O  significantly  changed 

If  significantly  changed,  also  rate  the  prior  images  where  requested. 

Otherwise,  rate  only  the  current  images. 

•  Size  of  distribution  of  asymmetric  breast  tissue 


Largest  diameter  in  CC  view 
Largest  diameter  in  oblique  view 


Current  Prior 

_ mm  _ 

_ mm  _ 


AT05 

AT06 


Confidence  regarding  the  presence  of  worrisome  calcifications  within  the  asymmetric  breast  tissue 


Current  012345 

definitely  NOT 
present 

Prior  01  2  3  4  5 


6  7  8  9  10  AT07 

definitely  present 


8 


10 


•  Confidence  regarding  presence  of  related  architectural  distortion 


Current 


1 


definitely  NOT 
present 

Prior  0  1 


9  10  AT09 

definitely  present 
9  10 
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•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chances  in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _  ATRA 

where;  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 
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Module  rv 


Reader  No. _ 

Case  No._ 
Finding  No., 


Architectural  Distortion  (AD) 


Relationship  to  Prior  Study 

•  This  architectural  distortion  finding  is:  ADI  1 

O  new 

O  not  significantly  changed 
O  significantly  changed 


If  significantly  changed,  also  rate  the  prior  images  where  requested. 
Otherwise,  rate  only  the  current  images. 


•  Confidence  that  the  architectural  distortion  is  related  to  prior  surgery 


Current  o  i 

2 

3 

4 

5 

6 

7 

8  9  10  AD06 

definitely  NOT 
related  to  prior  surgery 

Prior  0  i 

2 

3 

4 

5 

6 

7 

definitely 

related  to  prior  surgery 

8  9  10 

Confidence  regarding  the  presence  of  related  worrisome  calcifications 

Current  o  i 

2 

3  4  5  6  7  8 

9  10  AD07 

definitely  NOT 

definitely  present 

present 

Prior  0  i 

2 

3  4  5  6  7  8 

9  10 
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•  Confidence  regarding  the  presence  of  a  related  mass 


Current 

0  1 

2 

3 

4 

5 

6 

7 

8 

9  10  AD09 

definitely  NOT 

definitely  present 

present 

Prior 

0  1 

2 

3 

4 

5 

6 

7 

8 

9  10 

•  Confidence  regarding  the  presence  of  related  asymmetric  breast  tissue 


Current 

0  1 

2 

3 

4 

5 

6 

7 

8 

9  10  ADIO 

definitely  NOT 

definitely  present 

present 

Prior 

0  1 

2 

3 

4 

5 

6 

7 

8 

9  10 

Overall  Diagnostic  Tudement 


•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chances  in  100)  that  the  finding  is  indicative  of  malignancy: 

Rating  (0  to  100)  _ ^ _  ADRA 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 
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Module  V  Reader  No. _ 

Case  No. _ 

Finding  No. _ 

Regional  Calcifications  (Not-Definitely-Benign)  (RC) 
Relationship  to  Prior  Study 

•  This  regional  calcifications  finding  is:  RC21 

O  new 

O  not  significantly  changed 
O  significantly  changed 

If  significantly  changed,  aiso  rate  the  prior  images  where  requested. 

Otherwise,  rate  only  the  current  images. 

Element  Characteristics 


•  Size  of  largest  individual  element  (best  visual  estimate) 


Current  O  O  O  RC05 

less  than  0.5  mm  0.5  mm  to  1.0  mm  more  than  1.0  mm 

•  Variability  of  size  of  elements 


Current  0  1 

2 

3 

4 

5 

6 

7 

8 

9  10 

RC06 

low  variability 
of  size 

high  variability 
of  size 

•  Degree  to  which  the  elements  can  be  characterized  as  fine  linear 


Current  01234 

definitely  NONE  of  the 
elements  are  fine  linear 

Prior  0  12  3  4 


6  7  8  9  10  RC07 

at  least  one  or  two 
elements  definitely  are,  or 
several  probably  are,  fine  linear 

6  7  8  9  10 
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Calcifications  (Not-Definitely-Benign)  (RC)  -  cont, 


•  Degree  to  which  the  elements  can  be  characterized  as  branching 


Current  o 


10 


RC08 


definitely  NONE  of  the 
elements  are  branching 

Prior  0  12  3 


at  least  one  or  two 
elements  definitely  are,  or 
several  probably  are,  branching 

4  5  6  7  8  9  10 


•  Degree  to  which  the  elements  can  be  characterized  as  pleomorphic  (heterogeneous) 


Current  o 


2  3  4  5 


6  7  8  9  10 


RC09 


definitely  NONE  of  the 

elements  are  pleomorphic  (heterogeneous) 


at  least  one  or  two 

elements  definitely  are,  or  several  probably  ar* 
pleomorphic  (heterogeneous) 


Prior  0  1  2  3  4  5  6 

to  which  the  elements  can  be  characterized  as  punctate 

Current  0  1  2  3  4  5  6 

NONE  of  the  at  least  one  or  two 

elements  are  punctate 


7  8  9  10  • 

7  8  9  10  RCll 

elements  definitely  are,  or  several  probably  are 
punctate 


Relationship  to  Other  Aspects  of  This  Study 


•  Confidence  regarding  presence  of  related  architectural  distortion 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10 

RC19 

definitely  NOT 
present 

definitely 

present 

•  Confidence  regarding  presence  of  related  mass  or  asymmetric  breast  tissue 


Current  o  i 

2 

3 

4 

5 

6 

7 

8 

9  10 

RC20 

definitely  NOT 
present 

definitely 

present 
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Regional  Calcifications  (Not-Definitely-Benign)  (RC)  -  cont. 


Overall  Diagnostic  Tudgment 
•  Benign  vs.  Malignant 

Rate  the  likelihood  (as  the  number  of  chances  in  100)  that  the  finding  is  indicative  of 
malignancy: 

Rating  (0  to  100)  _ 

where:  0  =  certainly  benign  or  normal 
100  =  certainly  malignant 
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Appendix  B:  Sample  Reports  of  Automated  Report  Writer 


The  following  are  a  set  of  30  cases  selected  from  random  from  the  database  of  531  cases. 

Case  #118  2/11/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  5/31/1990.  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 

There  is  a  new  1 1  mm  irregular  mass  with  some  evidence  of  tissue  invasion  located  at  the  approximate  3  o’clock  middle 
position  of  the  right  breast.  A  small  portion  of  the  margin  is  indistinct  due  to  tissue  invasion.  About  half  of  the  margin  is 
obscured  by  glandular  tissue.  About  half  of  the  margin  is  clearly  circumscribed.  Microlobulation  is  present.  The  mass  may 
be  an  intramamary  node. 

There  is  a  85%  likelihood  that  the  mass  is  malignant. 

Case  #136_1  12/2/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  12/31/1992.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  new  25  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  2 
o'clock  posterior  position  of  the  left  breast.  About  half  of  the  margin  is  spiculated.  About  half  of  the  margin  is  indistinct  due 
to  tissue  invasion.  Microlobulation  is  present.  Architectural  distortions  are  evident.  Worrisome  and  benign  calcifications  are 
present. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case#143L  8/27/1990 

The  present  examination  is  compared  to  a  prior  mammogram  of  5/1 1/1989.  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 

There  is  a  6  mm  round/oval  mass  located  at  the  approximate  9  o’clock  middle  position  of  the  left  breast.  It  has  not  changed 
significantly  since  the  last  exam.  Some  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion  of  the  margin  is 
clearly  circumscribed.  The  moss  may  be  a  skin  lesion.  The  mass  may  be  an  intramamary  node. 

There  is  a  0%  likelihood  that  the  mass  is  malignant. 

Case  #145  3/13/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  6/25/1990.  The  breast  is  largely  glandular.  The  tissue  is 
very  dense. 
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There  is  a  new  7  mm  round/oval  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  9 
o’clock  middle  position  of  the  left  breast.  A  small  portion  of  the  margin  is  indistinct  due  to  tissue  invasion.  A  large  portion 
of  the  margin  is  obscured  by  glandular  tissue.  Some  of  the  margin  is  clearly  circumscribed. 

An  ultrasound  was  performed.  The  mass  appears  irregular  with  solid  contents.  The  mass  wall  is  indistinct.  The  posterior 
wall  of  the  mass  displays  shadowing. 

There  is  a  75%  likelihood  that  the  mass  is  malignant. 

Case  #151  7/6/1994 

The  present  examination  is  compared  to  a  prior  mammogram  of  7/30/1993.  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 

There  is  a  13  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  6  o'clock 
middle  position  of  the  right  breast.  It  has  not  changed  significantly  since  the  last  exam.  A  small  portion  of  the  margin  is 
indistinct  due  to  tissue  invasion.  A  large  portion  of  the  margin  is  obscured  by  glandular  tissue.  A  small  portion  of  the  margin 
is  clearly  circumscribed.  Microlobulation  is  present.  The  mass  may  be  an  intramamary  node. 

There  is  a  40%  likelihood  that  the  mass  is  malignant. 

Case  #164  10/30/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  4/19/1990.  The  breast  is  almost  entirely  fat. 
low  density. 

There  is  a  new  10  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the 
o'clock  middle  position  of  the  right  breast.  About  half  of  the  margin  is  spiculated.  About  half  of  the  margin 
to  tissue  invasion.  Microlobulation  is  present. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #189  8/11/1994 

The  present  examination  is  compared  to  a  prior  mammogram  of  12/8/1992.  The  breast  is  almost  entirely  glandular.  The 
tissue  is  moderately  dense. 

There  is  a  new  9  mm  round/oval  group  of  similar  masses  located  at  the  approximate  12  o'clock  middle  position  of  the  left 
breast.  Some  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion  of  the  margin  is  clearly  circumscribed. 
Microlobulation  is  present.  The  mass  appears  to  be  an  intramamary  node. 

There  is  a  5%  likelihood  that  the  mass  is  malignant. 


The  tissue  has  a 

approximate  11 
is  indistinct  due 
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Case  #201  1/6/1995 


The  present  examination  is  compared  to  a  prior  mammogram  of  7/16/1993.  The  breast  is  almost  entirely  fat.  The 
tissue  has  a  low  density. 

There  is  a  new  13  mm  lobular  mass  with  some  evidence  of  tissue  invasion  located  at  the  approximate  1  o’clock 
middle  position  of  the  right  breast.  A  large  portion  of  the  margin  is  obscured  by  glandular  tissue.  Some  of  the 
margin  is  clearly  circumscribed.  The  mass  may  be  an  intramamary  node. 

An  ultrasound  was  performed.  The  mass  appears  ellipsoid  with  solid  contents.  The  mass  wall  is  well- 
circumscribed.  The  posterior  wall  of  the  mass  is  iso-echoic. 

There  is  a  50%  likelihood  that  the  mass  is  malignant. 

Case  #223 J  12/16/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  7/19/1990.  The  breast  is  largely  glandular.  The 
tissue  is  moderately  dense. 

There  is  a  new  7  mm  irregular  group  of  similar  masses  with  some  evidence  of  spiculation  and  tissue  invasion  located 
in  the  left  breast.  A  large  portion  of  the  margin  is  spiculated.  A  small  portion  of  the  margin  is  indistinct  due  to 
tissue  invasion.  A  small  portion  of  the  margin  is  obscured  by  glandular  tissue.  Microlobulation  is  present. 
Architectural  distortions  are  evident. 

There  is  a  99%  likelihood  that  the  mass  is  malignant. 

Case  #230  10/11/1991 

The  present  examination  is  compared  to  a  prior  mammogram  of  8/14/1990.  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 

There  is  a  9  mm  round/oval  group  of  similar  masses  located  at  the  approximate  10  o'clock  posterior  position  of  the  right 
breast.  It  has  not  changed  significantly  since  the  last  exam.  Some  of  the  margin  is  obscured  by  glandular  tissue.  A  large 
portion  of  the  margin  is  clearly  circumscribed.  The  mass  is  definitely  an  intramamary  node. 

There  is  a  0%  likelihood  that  the  mass  is  malignant. 

Case  #254  1/5/1994 

The  present  examination  is  compared  to  a  prior  mammogram  of  12/17/1992.  The  breast  is  almost  entirely  fat.  The  tissue  has 
a  low  density. 

There  is  a  1 1  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasionanterior  of  the  right  breast.  It  has 
changed  significantly  since  the  last  exam.  About  half  of  the  margin  is  spiculated.  This  is  a  significant  increase  from  prior 
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examinations.  About  half  of  the  margin  is  indistinct  due  to  tissue  invasion.  This  is  a  significant  increase  from  prior 
examinations.  This  is  a  decrease  from  prior  examinations. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #272  11/30/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  1 1/19/1991 .  The  breast  is  almost  entirely  fat.  The  tissue  has 
a  low  density. 

There  is  a  15  mm  round/oval  mass  located  at  the  approximate  7  o’clock  middle  position  of  the  right  breast.  It  has  not  changed 
significantly  since  the  last  exam.  A  small  portion  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion  of  the 
margin  is  clearly  circumscribed.  The  mass  may  be  an  intramamary  node. 

There  is  a  5%  likelihood  that  the  mass  is  malignant. 

Case  #284  9/9/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  10/31/1990.  The  breast  is  largely  fat.  The  tissue  is 
moderately  dense. 

There  is  a  new  10  mm  lobular  mass  located  at  the  approximate  7  o’clock  anterior  position  of  the  left  breast.  A  small  portion 
of  the  margin  is  indistinct  due  to  tissue  invasion.  Some  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion  of  the 
margin  is  clearly  circumscribed.  The  mass  appears  to  be  a  skin  lesion. 

There  is  a  50%  likelihood  that  the  mass  is  malignant. 

Case  #293  5/7/1991 

The  present  examination  is  compared  to  a  prior  mammogram  of  10/2/1990.  The  breast  is  almost  entirely  glandular.  The 
tissue  is  very  dense. 

There  is  a  30  mm  round/oval  group  of  similar  masses  located  at  the  approximate  6  o’clock  middle  position  of  the  left  breast. 
It  has  not  changed  significantly  since  the  last  exam.  Some  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion  of 
the  margin  is  clearly  circumscribed. 

An  ultrasound  was  performed.  The  mass  appears  ellipsoid  with  cystic  contents.  The  mass  wall  is  well-circumscribed.  The 
posterior  wall  of  the  mass  displays  enhancement. 

There  is  a  0%  likelihood  that  the  mass  is  malignant. 

Case  #307  4/15/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  4/13/1990.  The  breast  is  largely  glandular.  The  tissue  is 
very  dense. 
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There  is  a  new  6  mm  round/oval  mass  located  in  the  right  breast.  Some  of  the  margin  is  obscured  by  glandular  tissue.  A 
large  portion  of  the  margin  is  clearly  circumscribed.  The  mass  is  definitely  an  intramamary  node. 

There  is  a  0%  likelihood  that  the  mass  is  malignant. 

Case  #328  5/1/1995 

The  present  examination  is  compared  to  a  prior  mammogram  of  5/3/1994.  The  breast  is  almost  entirely  fat.  The  tissue  is 
moderately  dense. 

There  is  a  1 1  mm  lobular  group  of  similar  masses  located  at  the  approximate  3  o’clock  posterior  position  of  the  left  breast.  It 
has  changed  significantly  since  the  last  exam.  A  small  portion  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion 
of  the  margin  is  clearly  circumscribed.  There  is  extensive  microlobulation  present.  The  mass  may  be  a  skin  lesion.  The 
mass  appears  to  be  an  intramamary  node. 

An  ultrasound  was  performed.  The  mass  appears  ellipsoid  with  solid  contents.  The  mass  wall  is  well-circumscribed.  The 
posterior  wall  of  the  mass  displays  enhancement. 

There  is  a  10%  likelihood  that  the  mass  is  malignant. 

Case  #350  5/9/1995 

The  present  examination  is  compared  to  a  prior  mammogram  of  5/25/1993.  The  breast  is  largely  fat.  The  tissue  is 
moderately  dense. 

There  is  a  12  mm  lobular  mass  with  some  evidence  of  tissue  invasion  located  at  the  approximate  3  o’clock  posterior  position 
of  the  left  breast.  It  has  not  changed  significantly  since  the  last  exam.  A  large  portion  of  the  margin  is  indistinct  due  to  tissue 
invasion.  About  half  of  the  margin  is  obscured  by  glandular  tissue.  Microlobulation  is  present. 

There  is  a  5%  likelihood  that  the  mass  is  malignant. 

Case  #357  12/9/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  9/17/1992.  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 

There  is  a  15  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  4  o'clock 
posterior  position  of  the  left  breast.  It  has  changed  significantly  since  the  last  exam.  A  large  portion  of  the  margin  is 
spiculated.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 
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Case  #380  8/10/1994 


The  present  examination  is  compared  to  a  prior  mammogram  of  6/17/1993.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  new  10  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  in  the  right  breast.  Some 
of  the  margin  is  spiculated.  A  large  portion  of  the  margin  is  indistinct  due  to  tissue  invasion.  Microlobulation  is  present. 

An  ultrasound  was  performed.  The  mass  appears  irregular  with  solid  contents.  The  mass  wall  is  indistinct.  The  posterior 
wall  of  the  mass  displays  shadowing. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #393  8/26/1994 

The  present  examination  is  compared  to  a  prior  mammogram  of  12/9/1992.  The  breast  is  almost  entirely  glandular.  The 
tissue  is  very  dense. 

There  is  a  new  15  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  12 
o'clock  posterior  position  of  the  right  breast.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion.  A  large  portion  of  the 
margin  is  obscured  by  glandular  tissue.  A  small  portion  of  the  margin  is  clearly  circumscribed.  Microlobulation  is  present. 
Architectural  distortions  are  evident.  Worrisome  calcifications  are  present. 

An  ultrasound  was  performed.  The  mass  appears  irregular  with  solid  contents.  The  mass  wall  is  irregular.  The  posterior 
wall  of  the  mass  displays  shadowing. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #398  8/16/1995 

The  present  examination  is  compared  to  a  prior  mammogram  of  8/25/1994.  The  breast  is  largely  glandular.  The  tissue  is 
very  dense. 

There  is  a  new  7  mm  lobular  mass  with  some  evidence  of  tissue  invasion  located  at  the  approximate  2  o'clock  position  of  the 
left  breast.  A  small  portion  of  the  margin  is  indistinct  due  to  tissue  invasion.  A  large  portion  of  the  margin  is  obscured  by 
glandular  tissue.  A  small  portion  of  the  margin  is  clearly  circumscribed.  The  mass  may  be  an  intramamary  node. 

An  ultrasound  was  performed.  The  mass  appears  ellipsoid  with  solid  contents.  The  mass  wall  is  well-circumscribed.  The 
posterior  wall  of  the  mass  displays  enhancement. 

There  is  a  25%  likelihood  that  the  mass  is  malignant. 
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Case  #412  6/19/1991 


The  present  examination  is  compared  to  a  prior  mammogram  of  6/11/1990.  The  breast  is  largely  fat.  The  tissue  is 
moderately  dense. 

There  is  a  17  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  3  o'clock 
anterior  position  of  the  right  breast.  It  has  not  changed  significantly  since  the  last  exam.  About  half  of  the  margin  is 
spiculated.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion.  Some  of  the  margin  is  obscured  by  glandular  tissue. 
Microlobulation  is  present.  Architectural  distortions  are  evident. 

There  is  a  60%  likelihood  that  the  mass  is  malignant. 

Case  #421  11/21/1994 

The  present  examination  is  compared  to  a  prior  mammogram  of  12/22/1993.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  new  21  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  8 
o’clock  position  of  the  right  breast.  A  small  portion  of  the  margin  is  spiculated.  About  half  of  the  margin  is  indistinct  due  to 
tissue  invasion.  About  half  of  the  margin  is  obscured  by  glandular  tissue.  Microlobulation  is  present.  Architectural 
distortions  are  evident. 

An  ultrasound  was  performed.  The  mass  appears  irregular  with  solid  contents.  The  mass  wall  is  irregular.  The  posterior 
wall  of  the  mass  displays  shadowing. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #441  9/29/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  7/9/1991 .  The  breast  is  largely  fat.  The  tissue  is  very  dense. 

There  is  a  7  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  1  o'clock 
posterior  position  of  the  right  breast.  It  has  changed  significantly  since  the  last  exam.  A  large  portion  of  the  margin  is 
spiculated.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion.  There  is  extensive  microlobulation  present.  Architectural 
distortions  are  evident. 

There  is  a  99%  likelihood  that  the  mass  is  malignant. 

Case  #445  9/15/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  3/19/1993,  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 
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There  is  a  1 1  mm  round/oval  group  of  similar  masses  located  at  the  approximate  8  o’clock  middle  position  of  the  right  breast. 
It  has  changed  significantly  since  the  last  exam.  A  small  portion  of  the  margin  is  obscured  by  glandular  tissue.  A  large 
portion  of  the  margin  is  clearly  circumscribed. 

There  is  a  5%  likelihood  that  the  mass  is  malignant. 

Case  #461  9/3/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  9/23/1991.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  new  25  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  4 
o'clock  posterior  position  of  the  left  breast.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion.  Microlobulation  is 
present.  Architectural  distortions  are  evident.  Worrisome  calcifications  are  present. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #467  2/10/1995 

The  present  examination  is  compared  to  a  prior  mammogram  of  10/14/1992.  The  breast  is  largely  fat.  The  tissue  is 
moderately  dense. 

There  is  a  new  15  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  9 
o’clock  middle  position  of  the  right  breast.  A  large  portion  of  the  margin  is  indistinct  due  to  tissue  invasion.  Some  of  the 
margin  is  clearly  circumscribed.  Architectural  distortions  are  evident.  Worrisome  calcifications  are  present. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 

Case  #479  3/24/1994 

The  present  examination  is  compared  to  a  prior  mammogram  of  3/23/1993.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  9  mm  lobular  mass  with  some  evidence  of  tissue  invasion  located  at  the  approximate  9  o'clock  middle  position  of 
the  left  breast.  It  has  changed  significantly  since  the  last  exam.  A  large  portion  of  the  margin  is  obscured  by  glandular  tissue. 
About  half  of  the  margin  is  clearly  circumscribed.  The  mass  may  be  a  skin  lesion.  The  mass  may  be  an  intramamary  node. 

An  ultrasound  was  performed.  The  mass  appears  ellipsoid  with  solid  contents.  The  mass  wall  is  well-circumscribed.  The 
posterior  wall  of  the  mass  is  iso-echoic. 

There  is  a  10%  likelihood  that  the  mass  is  malignant. 
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Case#492  4/15/1992 


The  present  examination  is  compared  to  a  prior  mammogram  of  2/27/1990.  The  breast  is  almost  entirely  fat.  The  tissue  has  a 
low  density. 

There  is  a  new  8  mm  round/oval  mass  located  at  the  approximate  8  o'clock  middle  position  of  the  right  breast.  A  small 
portion  of  the  margin  is  obscured  by  glandular  tissue.  A  large  portion  of  the  margin  is  clearly  circumscribed. 

There  is  a  2%  likelihood  that  the  mass  is  malignant. 

Case  #496  6/18/1992 

The  present  examination  is  compared  to  a  prior  mammogram  of  1 1/15/1990.  The  breast  is  almost  entirely  fat.  The  tissue  has 
a  low  density. 

There  is  a  6  mm  lobular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  12  o'clock 
position  of  the  right  breast.  It  has  changed  significantly  since  the  last  exam.  Some  of  the  margin  is  obscured  by  glandular 
tissue.  A  large  portion  of  the  margin  is  clearly  circumscribed.  Microlobulation  is  present. 

An  ultrasound  was  performed.  The  mass  appears  irregular  with  solid  contents.  The  mass  wall  is  indistinct.  The  posterior 
wall  of  the  mass  is  iso-echoic. 

There  is  a  15%  likelihood  that  the  mass  is  malignant. 
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Appendix  C:  Case  Descriptions 


The  following  six  reports  are  generated  from  the  same  case  read  by  six  different  radiologists.  Note  that  while  their  readings 
(and  the  reports  generated  from  them)  are  largely  similar,  there  are  some  differences. 

Case  #106  11/16/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  1/13/1992.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  7  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  2  o'clock 
middle  position  of  the  left  breast.  It  has  changed  significantly  since  the  last  exam.  Some  of  the  margin  is  spiculated.  This  is 
a  decrease  from  prior  examinations.  About  half  of  the  margin  is  indistinct  due  to  tissue  invasion.  This  is  a  significant 
increase  from  prior  examinations.  Some  of  the  margin  is  obscured  by  glandular  tissue.  Architectural  distortions  are  evident. 

There  is  a  90%  likelihood  that  the  mass  is  malignant. 

Case  #106  11/16/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  1/13/1992.  The  breast  is  largely  fat.  The  tissue  is 
moderately  dense. 

There  is  a  new  7  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  2 
o'clock  middle  position  of  the  left  breast.  About  half  of  the  margin  is  spiculated.  Some  of  the  margin  is  indistinct  due  to 
tissue  invasion.  Some  of  the  margin  is  obscured  by  glandular  tissue.  There  is  extensive  microlobulation  present.  The  mass 
may  be  a  skin  lesion.  The  mass  may  be  an  intramamary  node.  Architectural  distortions  are  evident. 

There  is  a  75%  likelihood  that  the  mass  is  malignant. 

Case  #106  11/16/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  1/13/1992.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  new  5  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  2 
o’clock  middle  position  of  the  left  breast.  About  half  of  the  margin  is  spiculated.  About  half  of  the  margin  is  indistinct  due  to 
tissue  invasion.  Microlobulation  is  present. 

There  is  a  100%  likelihood  that  the  mass  is  malignant. 
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Case  #106  11/16/1993 


The  present  examination  is  compared  to  a  prior  mammogram  of  1/13/1992.  The  breast  is  largely  glandular.  The  tissue  has  a 
low  density. 

There  is  a  8  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  2  o'clock 
middle  position  of  the  left  breast.  It  has  changed  significantly  since  the  last  exam.  About  half  of  the  margin  is  spiculated. 
This  is  a  significant  increase  from  prior  examinations.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion.  This  is  a 
significant  increase  from  prior  examinations.  Some  of  the  margin  is  obscured  by  glandular  tissue.  Microlobulation  is 
present.  Architectural  distortions  are  evident. 

There  is  a  98%  likelihood  that  the  mass  is  malignant. 

Case  #106  11/16/1993 

The  present  examination  is  compared  to  a  prior  mammogram  of  1/13/1992.  The  breast  is  largely  glandular.  The  tissue  is 
moderately  dense. 

There  is  a  7  mm  irregular  mass  with  some  evidence  of  spiculation  and  tissue  invasion  located  at  the  approximate  2  o’clock 
middle  position  of  the  left  breast.  It  has  not  changed  significantly  since  the  last  exam.  A  large  portion  of  the  margin  is 
spiculated.  Some  of  the  margin  is  indistinct  due  to  tissue  invasion.  A  small  portion  of  the  margin  is  obscured  by  glandular 
tissue.  There  is  extensive  microlobulation  present.  Architectural  distortions  are  evident. 

There  is  a  95%  likelihood  that  the  mass  is  malignant. 
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Appendix  D:  Sample  Case  As  Listed  in  Database 


The  following  is  an  example  of  the  fonnat  of  the  database  which  is  input  to  the  report  generator.  Each  item  is  a  pair  with  a 
keyword,  such  as  CASE,  RDR  (reader),  or  the  number  of  an  item  in  the  questionnaire,  followed  by  a  value,  such  as  the  case 
number,  reader’s  initials,  or  the  value  the  radiologist  entered  on  the  questionnaire  for  that  item.  Note  that  each  item  can  have 
a  current  reading  (_C)  and  a  previous  reading  (_P),  and  that  there  can  be  many  different  findings.  The  case  shown  here  is 
fairly  short,  since  there  is  only  one  finding  and  it  is  new,  so  most  items  only  have  a  current  but  no  previous  reading. 

CASE  118, 

RDR  PD, 

DOB  5/1/14, 

TRUTH  1, 

PATHM, 

DATE_C  2/1 1/1992, 

DATE_P  5/31/1990, 

INT_CP  1.7013698630137, 

AGE_C  77.8356164383562, 

AGE_P  76.1342465753425, 

OV02_C  20, 

OV02_P30, 

OV03_C2, 

OV03_P3, 

FINDING  mm, 

MMOIA 1, 

MM01_C  10, 

MM02_C  1, 

MM04_C  7, 

MM05_C  1, 

MM06_C9, 

MM07_C6, 

MM08_C11, 

MM09_C  6, 

MM10_C9, 

MM11_C8, 

MM12_C2, 

MM13A_C40, 

MM13B_C50, 

MM13C_C  10, 

MM13D_C  0, 

MM14_C  5, 

MM15_C0, 

MM16_C  1, 

MM17_C  1, 

MM18_C0, 

MM19_C0, 

MMULO, 

MMRA  85, 

FNDGNO, 

OV02_D-10, 

OV02_R  -5.87761674718197, 

OV03_D-1, 

OV03_R  -.587761674718196 
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Appendix  E:  Generation  Rules  for  Automated  Reports 


(defrule  (SREPORT 

>  SINTRO  SBLANKLINE  $FINDINGS  $BLANKLINE  $CONCLUSION 
SBLANKUNE)) 

(defrule  (SINTRO 

>  SCASE  STAB  SDATE  SBLANKLINE  SfflSTORY  SCOMPOSmON)) 

(defrule  (SCASE 

>  "Case  r(:VALUE  case) 
rCONDmON  ;EXISTS  CASE)) 

(defrule  (SDATE 

>(:  VALUE  DATE_C))) 

(defrule  (SHISTORY 

>  "The  present  examination  is  compared  to  a  prior  mammogram  of  " 
t’VAI  IIP  DATF  P'4 "  ” 

iCONDinON  :EXISTS  DATE_P)) 

(defrule  (SCOMPOSmON 

>  SGLANDULAR  SDENSITY)) 

(defrule  (SGLANDULAR 

>  "The  breast  is  almost  entirely  fat.  " 

:CONDmON  :BETWEEN  (:VALUE  OV02_C)  0  20 

>  "The  breast  is  largely  fat.  " 

:CONDmON  :BETWEEN  (:VALUE  OV02_C)  21  49 

>  "The  breast  is  largely  fibroglandular.  " 
iCONDinON  :BETWEEN  (: VALUE  OV02_C)  50  79 

>  "The  breast  is  almost  entirely  fibroglandular.  " 

:CONDITION  :BETWEEN  (: VALUE  OV02_C)  80  100)) 

(defrule  (SDENSITY 

>  "The  frbroglandular  tissue  has  a  low  density.  " 

:CONDITION  :BETWEEN  (:  VALUE  OV03_C)  0  3 

>  "The  fibroglandular  tissue  is  moderately  dense.  " 
rCONDmON  :BETWEEN  (iVALUE  OV03_C)  4  7 

>  "The  fibroglandular  tissue  is  very  dense.  " 

:CONDrnON  :BETWEEN  (:VALUE  OV03_C)  8  10)) 

(defhile  (SFINDINGS 

>  SMASS-FINDINGS 

rCONDinON  ;EQUAL  (: VALUE  FINDING)  mm)) 

(defiTile  (SMASS-FINDINGS 

>  SMASS-DESCRIPTION  SMASS-ATTRIBUTES  SMASS-ULTRASOUND)) 
(defhile  ( 

SMASS-DESCRIPnON 

>  "There  is  a  "  SMASS-NEW  SMASS-SIZE  SMASS-SHAPE 

SMASS-DISTRIBUnON  SMASS-DANGER  SMASS-LOCATION  SMASS-CHANGE)) 

(defrule  (SMASS-NEW 

>  "new  " 

iCONDmON  :EQUAL  (:VALUE  MMOIA)  1)) 
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(defhile  ($MASS-SIZE 

>  (:MAX-VALUE  MM06_C  MM07_C  MM08_C  MM09_C) "  mm  " 

:CONDmON  :EXISTS  (;MAX-VALUE  MM06_C  MM07_C  MM08_C  MM09_C))) 

(defhile  ($MASS-SHAPE 

>  "round/oval " 

rCONDmON  :LESS-OR-EQUAL  (:VALUE  MM10_C)  2 

>  "lobular " 

:CONDraON  :BETWEEN  (;  VALUE  MM10_C)  3  7 

>  "irregular  " 

:CONDmON  :GREATER-OR-EQUAL  (:VALUE  MMIO.C)  8)) 

(defhile  ($MASS-DISTRIBUnON 

>  "mass" 

:CONDmON  :EQUAL  (:  VALUE  MM02_C)  1 

>  "group  of  similar  masses" 

iCONOmON  :EQUAL  (:VALUE  MM02_C)  2)) 

(defhile  ($MASS-DANGER 

>  "  with  some  evidence  of  spiculation"  $MASS-DANGER2 
:CONDinON  :GREATER-OR-EQUAL  (:VALUE  MM12_C)  3 

>  "  with  some  evidence  of  tissue  invasion" 

:CONDinON  :GREATER-OR-EQUAL  (: VALUE  MMl  1_C)  3)) 

(defhile  ($MASS-DANGER2 

>  "  and  tissue  invasion" 

:CONDinON  :GREATER-OR-EQUAL  (:VALUE  MMl  1_C)  3)) 

(defhile  ($MASS-LOCATION 

>  $MASS-POSmON  $MASS-SIDE  ".  " 

:CONDmON  :BETWEEN  (:VALUE  MM03A)  1  14 
>$MASS-DEPTH$MASS-SIDE".  " 

:CONDmON  iBETWEEN  (:  VALUE  MM03B)  1  3 
>$ALT-MASS-SIDE".  " 

:CONDmON  :BETWEEN  (:VALUE  MM02A)  1  2 
>".  ")) 

(defhile  (SMASS-POSITION 

>  "  located  at  the  approximate  "  (:VALUE  MM03A) 

"  o’clock  "  $MASS-DEPTH  "position” 

:CONDrnON  :BETWEEN  (: VALUE  MM03A)  1  12 

>  "  located  at  the  central "  $MASS-DEPTH  "position" 
rCONDmON  :EQUAL  (: VALUE  MM03A)  13 

>  "  located  at  the  axillary  tail "  $MASS-DEPTH  "position" 

:CONDinON  :EQUAL  (:VALUE  MM03A)  14)) 

(defhile  ($MASS-DEPTH 

>  "anterior " 

rCONDinON  :EQUAL  (: VALUE  MM03B)  1 

>  "middle  depth " 

:CONDrnON  :EQUAL  (: VALUE  MM03B)  2 

>  "posterior " 

.CONDITION -EQUAL  (;  VALUE  MM03B)  3)) 

(defhile  ($MASS-SIDE 
> "  of  the  left  breast" 

:CONDinON  :EQUAL  (: VALUE  MM02A)  1 
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>  "  of  the  right  breast” 

iCONDmON  :EQUAL  (rVALUE  MM02A)  2)) 

(defrule  ($ALT-MASS-SIDE 

>  "  located  in  the  left  breast" 

:CONDrnON  :EQUAL  (:VALUE  MM02A)  1 

>  "  located  in  the  right  breast" 

:CONDrnON  :EQUAL  (:VALUE  MM02A)  2)) 

(defrule  (SMASS-ATTRIBUTES 

>$MASS-SPICULATED  $MASS-SPICULATED-CHANGE 
$MASS-INVASION  $MASS-INVASION-CHANGE 
$MASS-GLANDULAR  $MASS-GLANDULAR-CHANGE 
$MASS-CIRCUMSCRroED  $MASS-CIRCUMSCRIBED-CHANGE 
$MASS-MICROLOBULATION  $MASS-SKINLESION 
$MASS-INTRAMAMARY  $MASS-EXTRAS)) 

(defrule  ($MASS-INVASION 

>  "A  small  portion  of  the  margin  appears  to  be  indistinct" 

”  due  to  tissue  invasion.  " 

rCONDmON  :BETWEEN  (:VALUE  MM13C_C)  1  19 

>  "Some  of  the  margin  appears  to  be  indistinct  due  to  : 

"tissue  invasion.  " 

:CONDinON  ;BETWEEN  (:  VALUE  MM13C_C)  20  39 

>  "About  half  of  the  margin  appears  to  be  indistinct  due  to " 

"tissue  invasion.  " 

iCONDinON  :BETWEEN  (: VALUE  MM13C_C)  40  59 

>  "A  large  portion  of  the  margin  appears  to  be  indistinct  due  to" 

"  tissue  invasion.  " 

:CONDinON  :BETWEEN  (: VALUE  MM13C_C)  60  99 

>  "All  of  the  margin  appears  to  be  indistinct  due  to  " 

"tissue  invasion.  " 

iCONOmON  :EQUAL  (:VALUE  MM13C_C)  100)) 

(defrule  ($MASS-INVASION-CHANGE 

>  "This  is  a  significant  increase  from  the  prior  examination.  " 
:CONDrnON  :GREATER-THAN  (:VALUE  MM13C_D)  0 

>  "This  is  a  decrease  from  the  prior  examination.  " 

:CONDrnON  :LESS-THAN  (:VALUE  MM13C_D)  0)) 

(defrule  ($MASS-SPICULATED 

>  "A  small  portion  of  the  margin  is  spiculated,  " 

:CONDrnON  :BETWEEN  (:VALUE  MM13D_C)  1  19 

>  "Some  of  the  margin  is  spiculated.  " 
iCONDinON  :BETWEEN  C VALUE  MM13D_C)  20  39 

>  "About  half  of  the  margin  is  spiculated.  " 

:CONDITION  :BETWEEN  (: VALUE  MM13D_C)  40  59 

>  "A  large  portion  of  the  margin  is  spiculated.  " 

:CONDITION  iBETWEEN  (: VALUE  MM13D_C)  60  99 

>  "All  of  the  margin  is  spiculated.  " 

:CONDmON  :EQUAL  (: VALUE  MM13D_C)  100)) 

(defrule  ($MASS-SPICULATED-CHANGE 

>  "This  is  a  significant  increase  from  the  prior  examination.  " 
:CONDrnON  :GREATER-THAN  (;VALUE  MM13D_D)  0 

>  "This  is  a  decrease  from  the  prior  examination.  " 
rCONDmON  :LESS-THAN  (:VALUEMM13D_D)  0)) 
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(defhile  ($MASS-CIRCUMSCRIBED 

>  "A  small  portion  of  the  margin  is  clearly  circumscribed.  ” 
:CONDrnON  :BETWEEN  (:VALUE  MM13A_C)  1  19 

>  "Some  of  the  margin  is  clearly  circumscribed.  " 
rCONDmON  :BETWEEN  (:VALUE  MM13A_C)  20  39 

>  "About  half  of  the  margin  is  clearly  circumscribed.  " 

:CONDmON  :BETWEEN  (: VALUE  MM13A_C)  40  59 

>  "A  large  portion  of  the  margin  is  clearly  circumscribed.  " 
;CONDmON  iBETWEEN  (: VALUE  MM13A_C)  60  99 

>  "All  of  the  margin  is  clearly  circumscribed.  " 

:CONDmON  -.EQUAL  (:VALUE  MM13A_C)  100)) 

(defhile  ($MASS-GLANDULAR-CHANGE 

>  "This  is  a  significant  increase  from  the  prior  examination.  " 
:CONDmON  :GREATER-OR-EQUAL  (:VALUE  MM13A_D)  30 

>  "This  is  a  decrease  from  the  prior  exaiiunation.  " 

:CONDrnON  :LESS-OR-EQUAL  (: VALUE  MM13A_D)  -30)) 

(defhile  ($MASS-GLANDULAR 

>  "A  small  portion  of  the  margin  appears  to  be" 

"  obscured  by  glandular  tissue.  " 

iCONOmON  :BETWEEN  (rVALUE  MM13B_C)  1  19 

>  "Some  of  the  margin  appears  to  be  obscured  by  " 

"glandular  tissue.  " 

:CONDrnON  :BETWEEN  (:VALUE  MM13B_C)  20  39 

>  "About  half  of  the  margin  appears  to  be  obscured  by " 

"glandular  tissue.  " 

rCONOmON  :BETWEEN  (:VALUE  MM13B_C)  40  59 

>  "A  large  portion  of  the  margin  appears  to  be  obscured  by  " 
"glandular  tissue.  ” 

:CONDmON  :BETWEEN  (: VALUE  MM13B_C)  60  99 

>  "All  of  the  margin  appears  to  be  obscured  by  glandular  tissue.  " 
rCONDmON  :EQUAL  (:VALUE  MM13B_C)  100)) 

(defhile  (SMASS-GLANDULAR-CHANGE 

>  "This  is  a  significant  increase  from  the  prior  examination.  " 
:CONDrnON  :GREATER-OR-EQUAL  (: VALUE  MM13B_D)  30 

>  "This  is  a  decrease  from  the  prior  examination.  " 

:CONDmON  :LESS-OR-EQUAL  (:VALUE  MM13B_D)  -30)) 

(defhile  ($MASS-MICROLOBULATION 

>  "Microlobulation  is  present.  " 

:CONDmON  :BETWEEN  (: VALUE  MM14_C)  3  7 

>  "There  is  extensive  microlobulation  present.  " 

:CONDmON  :GREATER-OR-EQUAL  (:VALUE  MM14_C)  8)) 

(defhile  ($MASS-SKINLESION 

>  "The  mass  "  $MASS-SKINLESION-CONFIDENCE  "  a  skin  lesion. 
:CONDrnON  :GREATER-THAN  (:VALUEMM15_C)4)) 

(defhile  ($MASS-SKINLESION-CONFIDENCE 

>  "may  be" 

:CONDITION  :BETWEEN  (:VALUE  MM15_C)  5  8 

>  "appears  to  be" 

:CONDmON  :GREATER-THAN  (: VALUE  MM  15_C)  8)) 

(defrule  ($MASS-INTRAMAMARY 

>  "The  mass  "  $MASS-IhrnL«JvlAMARY-CONFIDENCE 
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"  an  intramamaiy  node.  " 

rCONDraON  :GREATER-THAN  (:VALUE  MM16_C)  4)) 

(defrule  ($MASS-nmAMAMARY-CONFIDENCE 

>  "may  be" 

:CONDrnON  :BETWEEN  (:VALUE  MM16_C)  5  8 

>  "appears  to  be" 

:CONDmON  :GREATER-THAN  (:VALUE  MM16_C)  8)) 

(defrule  ($MASS-EXTRAS 

>  "Architectural  distortion  is  evident.  "  $MASS-WC 
iCONDinON  :GREATER-OR-EQUAL  (;  VALUE  MM17_C)  5 
>$MASS-WC)) 

(defrule  ($MASS-WC 

>  "Worrisome"  $MASS-BC "  calcifications  are  present.  " 
:CONDmON  :GREATER-OR-EQUAL  (:VALUE  MM18_C)  5 

>  "Benign  calcifications  are  present.  " 

:CONDmON  :GREATER-OR-EQULA  (-.VALUE  MM19_C)  5)) 

(defrule  ($MASS-BC 

>  "  and  benign" 

:CONDmON  :GREATER-OR-EQUAL  (:VALUE  MM19_C)  5)) 

(defrule  ($MASS-ULTRASOUND 

>  $BLANKLINE  "An  ultrasound  was  performed.  " 
$MASS-US-SHAPE-CONTENTS  $MASS-US-APPEARANCE 
$MASS-US-RESPONSE 

.-CONDITION  :EQUAL  (; VALUE  MMUL)  1)) 

(defiule  ($MASS-US-SHAPE-CONTENTS 

>  $MASS-US-SHAPE  $MASS-US-CONTENTS-ALT 
:CONDrnON  :EXISTS  MMUL_1 

>  $MASS-US-CONTENTS)) 

(defrule  ($MASS-US-APPEARANCE 

>  "The  mass  wall  is  well-circumscribed.  " 

:CONDrnON  :EQUAL  (:  VALUE  MMUL_1)  1 

>  "The  mass  wall  is  indistinct.  " 
iCONDmON  :EQUAL  (:VALUE  MMUL_1)  2 

>  "The  mass  wall  is  irregular.  " 

:CONDmON  :EQUAL  (: VALUE  MMUL_1)  3)) 

(defrule  ($MASS-US-CONTENTS 

>  "The  contents  of  the  mass  are  solid.  " 

:CONDmON  :EQUAL  (:VALUE  MMUL_2)  1 

>  "The  contents  of  the  mass  are  indeterminate.  " 
rCONDinON  :EQUAL  (:VALUE  MMUL_2)  2 

>  "The  contents  of  the  mass  are  cystic.  " 
iCONDinON  :EQUAL  (: VALUE  MMUL_2)  3)) 

(defrule  ($MASS-US-CONTENTS-ALT 

>  "with  solid  contents.  " 

iCONDmON  :EQUAL  (:VALUE  MMUL_2)  1 

>  "with  indeterminate  contents.  " 

:CONDrnON  :EQUAL  (:VALUE  MMUL_2)  2 

>  "with  cystic  contents.  " 

:CONDrnON  :EQUAL  (:VALUE  MMUL_2)  3 
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>".")) 

(defhile  ($MASS-US-RESPONSE 

>  "The  posterior  wall  of  the  mass  displays  enhancement.  " 
:CONDmON  :EQUAL  (:VALUE  MMUL_3)  1 

>  "The  posterior  wall  of  the  mass  is  iso-echoic.  " 

:CONDmON  :EQUAL  (:VALUE  MMUL_3)  2 

>  "The  posterior  wall  of  die  mass  displays  shadowing.  " 
tCONDmON  :EQUAL  (: VALUE  MMUL_3)  3)) 

(defrule  ($MASS-US-SHAPE 

>  "The  mass  appears  round  " 

:CONDmON  :EQUAL  (rVALUE  MMUL_4)  1 

>  "The  mass  appears  ellipsoid  " 

:CONDmON  :EQUAL  (:VALUE  MMUL_4)  2 

>  "The  mass  appears  irregular  " 
iCONOmON  :EQUAL  (:VALUE  MMUL_4)  3)) 

(defrule  ($CONCLUSION 

>  $MASS-CONCLUSION 

:CONDmON  :EQUAL  (:VALUE  FINDING)  mm)) 

(defrule  ($MASS-CONCLUSION 

>  $MASS-MAUGNANT)) 

(defrule  ($MASS-COMPARE 

>  "This  is  a  new  finding.  "  $NEWLINE 
:CONDmON  :EQUAL  (:VALUE  MMOl A)  1 

>  $MASS-CHANGE  SNEWLINE 
:CONDmON  :EQUAL  (:  VALUE  MMOl  A)  0)) 

(defrule  ( $MASS-CHANGE 

>  "It  has  changed  significantly  since  the  last  exam.  " 

.  :CONDrnON  :EQUAL  (:VALUE  MMOIB)  1 

>  "It  has  not  changed  significantly  since  the  last  exam.  " 
:CONDrnON  :EQUAL  (: VALUE  MMOIB)  0)) 

(defhile  ($MASS-MALIGNANT 

>  "Impression:  There  is  a  "  (:VALUE  MMRA)  "%  likelihood" 

"  that  the  mass  is  malignant.  "  SNEWLINE 

:CONDrnON  :GREATER-OR-EQUAL  (: VALUE  MMRA)  0)) 

(defiule  (SNEWLINE 

>  "-%")) 

(defrule  (SBLANKLINE 

>  SNEWLINE  SNEWLINE)) 

(defrule  (STAB 
>"  ")))) 
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Appendix  F:  Description  of  the  Rules  for  Automated  Reports 

The  following  is  a  more  compact  description  of  the  rules.  We  begin  with  the  beginning  “nonterminal”,  $REPORT  (note  all 
non-terminals  begin  with  the  special  symbol  $  to  more  easily  distinguish  them  from  terminal  symbols).  Each  nonterminal  is 
expanded  using  the  rules  shown  in  Appendix  E  until  “terminals”  are  reached.  When  there  are  multiple  terms  in  the 
expansion,  they  are  shown  at  the  same  level  of  indentation  in  the  list  (e.g.,  the  $INTRO  is  expanded  to  $CASE,  $DATE, 
SfflSTORY  and  SCOMPOSITION  and  $COMPOSmON  expands  to  SGLANDULAR  followed  by  SDENSITY) 

The  terminals  are  the  words  that  are  printed  in  the  report,  and  thus  they  don’t  expand  any  further.  When  there  are  options  in 
the  rules  (that  is  there  is  more  than  one  expansion  of  a  non-terminal),  each  option  is  preceded  by  a  >  symbol.  If  a  value  is  to 
be  read  from  the  database  (such  as  the  date  or  case  number),  that  item  is  in  all  capitals  (e.g.,  DATE_C,  CASE). 


SREPORT 
SINTRO 
$CASE 
Case  CASE 
$DATE 
DATE_C 
SfflSTORY 

The  present  examination  is  compared  to  a  prior  mammogram  of  DATE_P. 
SCOMPOSITION 
SGLANDULAR 
>The  breast  is  almost  entirely  fat. 

>The  breast  is  largely  fat. 

>The  breast  is  largely  glandular. 

>The  breast  is  almost  entirely  glandular. 

SDENSITY 

>The  tissue  has  a  low  density. 

>The  tissue  is  moderately  dense. 

>The  tissue  is  very  dense. 

SFINDINGS 
SMASS-FINDINGS 
SMASS-DESCRIPnON 
There  is  a 
SMASS-NEW 
new 

SMASS-SIZE 

[MAX- VALUE  MM06_C  MM07_C  MM08_C  MM09_C]  mm 
SMASS-SHAPE 
>round/oval 
>lobular 
>irregular 

SMASS-DISTRIBUnON 

>mass 

>group  of  similar  masses 
$MASS-DANGER 

>  with  some  evidence  of  spiculation 
$MASS-DANGER2 

and  tissue  invasion 

>  with  some  evidence  of  tissue  invasion 
$MASS-LOCATION 

>$MASS-POSmON 

>  located  at  the  approximate  MM03A  o'clock 
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$MASS-DEPTH 

>anterior 

>middle 

>posterior 

position 

>  located  at  the  central 
$MASS-DEPTH 

>anterior 

>iniddle 

>posterior 

position 

>  located  at  the  axillary  tail 
$MASS-DEPTH 

>anterior 

>middle 

>posterior 

position 

$MASS-SIDE 

>  of  the  left  breast 

>  of  the  right  breast 

>$MASS-DEPTH 

>anterior 

>iniddle 

>posterior 

$MASS-SIDE 

>  of  the  left  breast 

>  of  the  right  breast 

>$ALT-MASS-SIDE 

>  located  in  the  left  breast 

>  located  in  the  right  breast 


>. 

$MASS-CHANGE 

>It  has  changed  significantly  since  the  last  exam. 

>It  has  not  changed  significantly  since  the  last  exam. 
SMASS-ATTRIBUTES 
$MASS-SPICULATED 
>A  small  portion  of  the  margin  is  spiculated. 

>Some  of  the  margin  is  spiculated. 

>About  half  of  the  margin  is  spiculated. 

>A  large  portion  of  the  margin  is  spiculated. 

>A11  of  the  margin  is  spiculated. 
$MASS-SPICULATED-CHANGE 
>This  is  a  significant  increase  from  prior  examinations. 

>This  is  a  decrease  from  prior  examinations. 

$MASS-INVASION 

>A  small  portion  of  the  margin  is  indistinct  due  to  tissue  invasion. 
>Some  of  the  margin  is  indistinct  due  to  tissue  invasion. 

>About  half  of  the  margin  is  indistinct  due  to  tissue  invasion. 

>A  large  portion  of  the  margin  is  indistinct  due  to  tissue  invasion. 
>A11  of  the  margin  is  indistinct  due  to  tissue  invasion. 
$MASS-INVASION-CHANGE 
>This  is  a  significant  increase  from  prior  examinations. 

>This  is  a  decrease  from  prior  examinations. 
$MASS-GLANDULAR 

>A  small  portion  of  the  margin  is  obscured  by  glandular  tissue. 
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>Some  of  the  margin  is  obscured  by  glandular  tissue. 

>About  half  of  the  margin  is  obscured  by  glandular  tissue. 

>A  large  portion  of  the  margin  is  obscured  by  glandular  tissue. 
>A11  of  the  margin  is  obscured  by  glandular  tissue. 
$MASS-GLANDULAR-CHANGE 
>This  is  a  significant  increase  from  prior  examinations. 

>This  is  a  decrease  from  prior  examinations. 
$MASSCIRCUMSCRIBED 
>A  small  portion  of  the  margin  is  clearly  circumscribed. 
>Some  of  the  margin  is  clearly  circumscribed. 

>About  half  of  the  margin  is  clearly  circumscribed. 

>A  large  portion  of  the  margin  is  clearly  circumscribed. 

>AI1  of  the  margin  is  clearly  circumscribed. 
$MASS-CIRCUMSCRIBED-CHANGE 
$MASS-MICROLOBULATION 
>Microlobulation  is  present. 

>There  is  extensive  microlobulation  present. 
$MASS-SKINLESION 
The  mass 

$MASS-SKINLESION-CONFIDENCE 
>may  be 
>appears  to  be 
>is  definitely 
a  skin  lesion. 

$MASS-INTRAMAMARY 
The  mass 

$MASS-IhrniAMAMARY-CONFro 
>may  be 
>appears  to  be 
>is  definitely 
an  intramamary  node. 

$MASS-EXTRAS 

>Architectural  distortions  are  evident. 

$MASS-WC 
>Worrisome 
$MASS-BC 
and  benign 

calcifications  are  present. 

>Benign  calcifications  are  present. 

>$MASS-WC 
>Worrisome 
$MASS-BC 
and  benign 

calcifications  are  present. 

>Benign  calcifications  are  present. 

$MASS-ULTRASOUND 
An  ultrasound  was  performed. 

$MASS-US-SHAPE-CONTENTS 
>$MASS-US-SHAPE 
>The  mass  appears  round 
>The  mass  appears  ellipsoid 
>The  mass  appears  irregular 
$MASS-US-CONTENTS-ALT 
>with  solid  contents. 

>with  indeterminate  contents. 

>with  cystic  contents. 

>. 

>$MASS-US-CONTENTS 
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>The  contents  of  the  mass  are  solid. 

>The  contents  of  the  mass  are  indeterminate. 

>The  contents  of  the  mass  are  cystic. 
$MASS-US-APPEARANCE 
>The  mass  wall  is  well-circumscribed. 

>The  mass  wall  is  indistinct. 

>The  mass  wall  is  irregular. 

$MASS-US-RESPONSE 

>The  posterior  wall  of  the  mass  displays  enhancement. 
>The  posterior  wall  of  the  mass  is  iso-echoic. 

>The  posterior  wall  of  the  mass  displays  shadowing. 
SCONCLUSION 
$MASS-CONCLUSION 
SMASS-MALIGNANT 

There  is  a  MMRA%  likelihood  that  the  mass  is  malignant. 
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Appendix  G:  Instructions  for  Mammogram  T^t  Readers 

Instructions  for  Mammogram  Test  Readers 
Harvard  Pilgrim  Health  Care 
July,  1997 

Greetings!  On  this  page,  we  researchers  from  BBN  Corporation  and  the  Brigham  and 
Women’s  Hospital  describe  your  task  as  you  read  a  set  of  150  proven  mammograms.  The 
research  is  supported  by  the  U.  S.  Army  Medical  Research  and  Materiel  Command. 

Your  first  time  through  the  set  (during  July  and  August)  is  a  “baseline”  condition,  similar 
to  readings  in  your  usual  practice.  A  second  condition  (during  the  winter)  provides  “enhanced” 
readings,  with  a  decision  aid  to  be  described  later.  Your  personal  interest  in  the  baseline 
condition  may  derive  from  your  receiving  a  confidential  report  of  your  accuracy  relative  to  the 
(anonymous)  performances  of  your  colleagues.  In  the  enhanced  condition,  our  previous 
experience  suggests  that  everyone’s  accuracy  will  be  noticeably  improved  by  the  decision  aid. 

One  distinctive  aspect  of  your  baseline  readings  is  that  you  will  be  reporting  a  “probability 
of  malignancy”  for  each  case  rather  than  making  the  usual  diagnosis  and  treatment 
recommendation.  Thus,  you  will  be  rating  the  likelihood  of  malignancy  on  a  scale  from  0  to 
100.  In  short,  you  will  express  the  chances  out  of  100  cases  that  a  case  exactly  like  the  one  at 
hand  will  be  shown  by  biopsy  to  be  malignant. 

A  second  distinctive  aspect  of  your  readings  is  that  the  set  of  150  cases  contains  more 
malignant  cases  than  you  would  experience  in  your  usual  practice.  Specifically,  approximately 
one-third  (i.e.,  about  50)  of  the  cases  are  malignant.  Beyond  that,  approximately  one-third  of 
the  cases  contain  a  proven  benign  lesion.  The  remaining  one-third  are  defined  as  “suspicious 
normals”  —  i.e.,  cases  that  were  recommended  for  immediate  further  workup  or  accelerated 
follow-up  manunograms,  but  were  shown  by  subsequent  examinations  over  time  to  be  non- 
malignant.  Hence,  you  may  assign  somewhat  higher  probabilities  of  malignancy  to  these  test 
cases  than  you  would  to  a  consecutive  set  of  150  cases  in  usual  practice,  in  which  the  number 
of  proven  malignancies  might  be,  say,  3  to  5.  We  perform  a  statistical  analysis  of  reading 
performance  —  so  called  ROC  analysis  —  that  gives  a  measure  of  accuracy  that  is  unaffected  by 
the  proportion  of  malignant  cases  in  a  test  set.  This  measure  is  also  unaffected  by  the  specific 
way  in  which  the  probability  scale  is  used  by  any  given  reader,  as  long  as  nearly  the  full  scale 
is  used,  and  is  used  in  a  self-consistent  manner. 

Please  ask  us  any  questions  you  may  have  now.  After  a  few  practice  cases,  we  hope  that 
you  will  take  a  reasonably  constant  approach  to  all  remaining  cases. 
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